{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Symbol Tutorial\n",
    "\n",
    "Besides the tensor computation interface [NDArray](./ndarray.ipynb), another main object in MXNet is the `Symbol` provided by `mxnet.symbol`, or `mxnet.sym` for short. A symbol represents a multi-output symbolic expression. They are composited by operators, such as simple matrix operations (e.g. “+”), or a neural network layer (e.g. convolution layer). An operator can take several input variables, produce more than one output variables, and have internal state variables. A variable can be either free, which we can bind with value later, or an output of another symbol. \n",
    "\n",
    "## Symbol Composition \n",
    "### Basic Operators\n",
    "The following example composites a simple expression `a+b`. We first create the placeholders `a` and `b` with names using `mx.sym.Variable`, and then construct the desired symbol by using the operator `+`. When the string name is not given during creating, MXNet will automatically generate a unique name for the symbol, which is the case for `c`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Symbol a>, <Symbol b>, <Symbol _plus0>)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import mxnet as mx\n",
    "a = mx.sym.Variable('a')\n",
    "b = mx.sym.Variable('b')\n",
    "c = a + b\n",
    "(a, b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most `NDArray` operators can be applied to `Symbol`, for example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"442pt\"\n",
       " viewBox=\"0.00 0.00 214.00 442.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 438)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-438 210,-438 210,4 -4,4\"/>\n",
       "<!-- a -->\n",
       "<g id=\"node1\" class=\"node\"><title>a</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "</g>\n",
       "<!-- b -->\n",
       "<g id=\"node2\" class=\"node\"><title>b</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "</g>\n",
       "<!-- _mul0 -->\n",
       "<g id=\"node3\" class=\"node\"><title>_mul0</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"94,-152 -7.10543e-15,-152 -7.10543e-15,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-119.3\" font-family=\"Times,serif\" font-size=\"14.00\">_mul</text>\n",
       "</g>\n",
       "<!-- _mul0&#45;&gt;a -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>_mul0&#45;&gt;a</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-83.7443C47,-75.2043 47,-66.2977 47,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-93.8971 42.5001,-83.897 47,-88.8971 47.0001,-83.8971 47.0001,-83.8971 47.0001,-83.8971 47,-88.8971 51.5001,-83.8971 47,-93.8971 47,-93.8971\"/>\n",
       "</g>\n",
       "<!-- _mul0&#45;&gt;b -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>_mul0&#45;&gt;b</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M89.3281,-87.2305C103.403,-75.6693 118.704,-63.1001 131.373,-52.6936\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"81.2123,-93.8971 86.0832,-84.0724 85.0759,-90.7234 88.9395,-87.5496 88.9395,-87.5496 88.9395,-87.5496 85.0759,-90.7234 91.7959,-91.0269 81.2123,-93.8971 81.2123,-93.8971\"/>\n",
       "</g>\n",
       "<!-- dot0 -->\n",
       "<g id=\"node4\" class=\"node\"><title>dot0</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"206,-152 112,-152 112,-94 206,-94 206,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-119.3\" font-family=\"Times,serif\" font-size=\"14.00\">dot</text>\n",
       "</g>\n",
       "<!-- dot0&#45;&gt;a -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>dot0&#45;&gt;a</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M116.672,-87.2305C102.597,-75.6693 87.2958,-63.1001 74.627,-52.6936\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"124.788,-93.8971 114.204,-91.0269 120.924,-90.7234 117.06,-87.5496 117.06,-87.5496 117.06,-87.5496 120.924,-90.7234 119.917,-84.0724 124.788,-93.8971 124.788,-93.8971\"/>\n",
       "</g>\n",
       "<!-- dot0&#45;&gt;b -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>dot0&#45;&gt;b</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M159,-83.7443C159,-75.2043 159,-66.2977 159,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"159,-93.8971 154.5,-83.897 159,-88.8971 159,-83.8971 159,-83.8971 159,-83.8971 159,-88.8971 163.5,-83.8971 159,-93.8971 159,-93.8971\"/>\n",
       "</g>\n",
       "<!-- _plus1 -->\n",
       "<g id=\"node5\" class=\"node\"><title>_plus1</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"150,-246 56,-246 56,-188 150,-188 150,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-213.3\" font-family=\"Times,serif\" font-size=\"14.00\">elemwise_add</text>\n",
       "</g>\n",
       "<!-- _plus1&#45;&gt;_mul0 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>_plus1&#45;&gt;_mul0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M80.5686,-179.148C75.113,-170.186 69.3635,-160.74 64.1943,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"85.8939,-187.897 76.8505,-181.695 83.2941,-183.626 80.6944,-179.355 80.6944,-179.355 80.6944,-179.355 83.2941,-183.626 84.5383,-177.015 85.8939,-187.897 85.8939,-187.897\"/>\n",
       "</g>\n",
       "<!-- _plus1&#45;&gt;dot0 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>_plus1&#45;&gt;dot0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.431,-179.148C130.887,-170.186 136.637,-160.74 141.806,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"120.106,-187.897 121.462,-177.015 122.706,-183.626 125.306,-179.355 125.306,-179.355 125.306,-179.355 122.706,-183.626 129.15,-181.695 120.106,-187.897 120.106,-187.897\"/>\n",
       "</g>\n",
       "<!-- reshape0 -->\n",
       "<g id=\"node6\" class=\"node\"><title>reshape0</title>\n",
       "<polygon fill=\"#fdb462\" stroke=\"black\" points=\"150,-340 56,-340 56,-282 150,-282 150,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-307.3\" font-family=\"Times,serif\" font-size=\"14.00\">Reshape</text>\n",
       "</g>\n",
       "<!-- reshape0&#45;&gt;_plus1 -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>reshape0&#45;&gt;_plus1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M103,-271.744C103,-263.204 103,-254.298 103,-246.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103,-281.897 98.5001,-271.897 103,-276.897 103,-271.897 103,-271.897 103,-271.897 103,-276.897 107.5,-271.897 103,-281.897 103,-281.897\"/>\n",
       "</g>\n",
       "<!-- broadcast_to0 -->\n",
       "<g id=\"node7\" class=\"node\"><title>broadcast_to0</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"150,-434 56,-434 56,-376 150,-376 150,-434\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-401.3\" font-family=\"Times,serif\" font-size=\"14.00\">broadcast_to</text>\n",
       "</g>\n",
       "<!-- broadcast_to0&#45;&gt;reshape0 -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>broadcast_to0&#45;&gt;reshape0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M103,-365.744C103,-357.204 103,-348.298 103,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103,-375.897 98.5001,-365.897 103,-370.897 103,-365.897 103,-365.897 103,-365.897 103,-370.897 107.5,-365.897 103,-375.897 103,-375.897\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f053015d750>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# elemental wise times\n",
    "d = a * b  \n",
    "# matrix multiplication\n",
    "e = mx.sym.dot(a, b)   \n",
    "# reshape\n",
    "f = mx.sym.Reshape(d+e, shape=(1,4))  \n",
    "# broadcast\n",
    "g = mx.sym.broadcast_to(f, shape=(2,4))  \n",
    "mx.viz.plot_network(symbol=g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Neural Networks\n",
    "\n",
    "Besides the basic operators, `Symbol` has a rich set of neural network layers. The following codes construct a two layer fully connected neural work and then visualize the structure by given the input data shape. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"506pt\"\n",
       " viewBox=\"0.00 0.00 214.00 506.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 502)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-502 210,-502 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-168 -7.10543e-15,-168 -7.10543e-15,-110 94,-110 94,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-99.8131C47,-86.1516 47,-71.0092 47,-58.3283\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-109.906 42.5001,-99.9062 47,-104.906 47.0001,-99.9062 47.0001,-99.9062 47.0001,-99.9062 47,-104.906 51.5001,-99.9062 47,-109.906 47,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">200</text>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node3\" class=\"node\"><title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-278 -7.10543e-15,-278 -7.10543e-15,-220 94,-220 94,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-252.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-237.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-209.813C47,-196.152 47,-181.009 47,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-219.906 42.5001,-209.906 47,-214.906 47.0001,-209.906 47.0001,-209.906 47.0001,-209.906 47,-214.906 51.5001,-209.906 47,-219.906 47,-219.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node4\" class=\"node\"><title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-388 -7.10543e-15,-388 -7.10543e-15,-330 94,-330 94,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-319.813C47,-306.152 47,-291.009 47,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-329.906 42.5001,-319.906 47,-324.906 47.0001,-319.906 47.0001,-319.906 47.0001,-319.906 47,-324.906 51.5001,-319.906 47,-329.906 47,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- out_label -->\n",
       "<g id=\"node5\" class=\"node\"><title>out_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-359\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-355.3\" font-family=\"Times,serif\" font-size=\"14.00\">out_label</text>\n",
       "</g>\n",
       "<!-- out -->\n",
       "<g id=\"node6\" class=\"node\"><title>out</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"170,-498 76,-498 76,-440 170,-440 170,-498\"/>\n",
       "<text text-anchor=\"middle\" x=\"123\" y=\"-465.3\" font-family=\"Times,serif\" font-size=\"14.00\">SoftmaxOutput</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;fc2 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>out&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M97.2538,-431.413C87.3525,-417.343 76.2083,-401.506 66.9347,-388.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103.23,-439.906 93.7952,-434.318 100.353,-435.817 97.4753,-431.728 97.4753,-431.728 97.4753,-431.728 100.353,-435.817 101.155,-429.138 103.23,-439.906 103.23,-439.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"97\" y=\"-410.3\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;out_label -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>out&#45;&gt;out_label</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M135.648,-430.057C140.313,-416.062 145.505,-400.485 149.807,-387.579\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"132.365,-439.906 131.258,-428.996 133.946,-435.163 135.527,-430.419 135.527,-430.419 135.527,-430.419 133.946,-435.163 139.796,-431.842 132.365,-439.906 132.365,-439.906\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f053015df50>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Output may vary\n",
    "net = mx.sym.Variable('data')\n",
    "net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)\n",
    "net = mx.sym.Activation(data=net, name='relu1', act_type=\"relu\")\n",
    "net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=10)\n",
    "net = mx.sym.SoftmaxOutput(data=net, name='out')\n",
    "mx.viz.plot_network(net, shape={'data':(100,200)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modulelized Construction for Deep Networks\n",
    "For deep networks, such as the Google Inception, constructing layer by layer is painful given the large number of layers. For these networks, we often modularize the construction. Take the Google Inception as an example, we can first define a factory function to chain the convolution layer, batch normalization layer, and Relu activation layer together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"550pt\" height=\"396pt\"\n",
       " viewBox=\"0.00 0.00 550.00 396.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 392)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-392 546,-392 546,4 -4,4\"/>\n",
       "<!-- Previos Output -->\n",
       "<g id=\"node1\" class=\"node\"><title>Previos Output</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">Previos Output</text>\n",
       "</g>\n",
       "<!-- conv_None -->\n",
       "<g id=\"node2\" class=\"node\"><title>conv_None</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-168 -7.10543e-15,-168 -7.10543e-15,-110 94,-110 94,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">7x7/2, 64</text>\n",
       "</g>\n",
       "<!-- conv_None&#45;&gt;Previos Output -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>conv_None&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-99.8131C47,-86.1516 47,-71.0092 47,-58.3283\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-109.906 42.5001,-99.9062 47,-104.906 47.0001,-99.9062 47.0001,-99.9062 47.0001,-99.9062 47,-104.906 51.5001,-99.9062 47,-109.906 47,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"71\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_None_gamma -->\n",
       "<g id=\"node3\" class=\"node\"><title>bn_None_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_gamma</text>\n",
       "</g>\n",
       "<!-- bn_None_beta -->\n",
       "<g id=\"node4\" class=\"node\"><title>bn_None_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"271\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"271\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_beta</text>\n",
       "</g>\n",
       "<!-- bn_None_moving_mean -->\n",
       "<g id=\"node5\" class=\"node\"><title>bn_None_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"383\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"383\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_None_moving_var -->\n",
       "<g id=\"node6\" class=\"node\"><title>bn_None_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"495\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"495\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_None -->\n",
       "<g id=\"node7\" class=\"node\"><title>bn_None</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"318,-278 224,-278 224,-220 318,-220 318,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"271\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;conv_None -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>bn_None&#45;&gt;conv_None</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M214.357,-228.844C193.753,-221.259 170.501,-211.968 150,-202 129.54,-192.052 107.83,-179.335 89.6442,-168.011\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"223.926,-232.316 212.99,-233.135 219.225,-230.61 214.525,-228.905 214.525,-228.905 214.525,-228.905 219.225,-230.61 216.06,-224.675 223.926,-232.316 223.926,-232.316\"/>\n",
       "<text text-anchor=\"middle\" x=\"177.5\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x11x11</text>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_gamma -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M234.454,-212.759C217.87,-196.768 198.688,-178.27 183.814,-163.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"241.866,-219.906 231.544,-216.204 238.266,-216.436 234.667,-212.965 234.667,-212.965 234.667,-212.965 238.266,-216.436 237.791,-209.726 241.866,-219.906 241.866,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_beta -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M271,-209.813C271,-196.152 271,-181.009 271,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"271,-219.906 266.5,-209.906 271,-214.906 271,-209.906 271,-209.906 271,-209.906 271,-214.906 275.5,-209.906 271,-219.906 271,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_moving_mean -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M307.546,-212.759C324.13,-196.768 343.312,-178.27 358.186,-163.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"300.134,-219.906 304.209,-209.726 303.734,-216.436 307.333,-212.965 307.333,-212.965 307.333,-212.965 303.734,-216.436 310.456,-216.204 300.134,-219.906 300.134,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_moving_var -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M327.573,-221.392C360.219,-205.955 402.007,-186.054 439,-168 445.545,-164.806 452.5,-161.366 459.202,-158.027\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"318.186,-225.827 325.306,-217.486 322.707,-223.691 327.228,-221.555 327.228,-221.555 327.228,-221.555 322.707,-223.691 329.15,-225.624 318.186,-225.827 318.186,-225.827\"/>\n",
       "</g>\n",
       "<!-- relu_None -->\n",
       "<g id=\"node8\" class=\"node\"><title>relu_None</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"318,-388 224,-388 224,-330 318,-330 318,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"271\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"271\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_None&#45;&gt;bn_None -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>relu_None&#45;&gt;bn_None</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M271,-319.813C271,-306.152 271,-291.009 271,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"271,-329.906 266.5,-319.906 271,-324.906 271,-319.906 271,-319.906 271,-319.906 271,-324.906 275.5,-319.906 271,-329.906 271,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"298.5\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x11x11</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f05a67c5dd0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Output may vary\n",
    "def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0), name=None, suffix=''):\n",
    "    conv = mx.symbol.Convolution(data=data, num_filter=num_filter, kernel=kernel, stride=stride, pad=pad, name='conv_%s%s' %(name, suffix))\n",
    "    bn = mx.symbol.BatchNorm(data=conv, name='bn_%s%s' %(name, suffix))\n",
    "    act = mx.symbol.Activation(data=bn, act_type='relu', name='relu_%s%s' %(name, suffix))\n",
    "    return act\n",
    "prev = mx.symbol.Variable(name=\"Previos Output\")\n",
    "conv_comp = ConvFactory(data=prev, num_filter=64, kernel=(7,7), stride=(2, 2))\n",
    "shape = {\"Previos Output\" : (128, 3, 28, 28)}\n",
    "mx.viz.plot_network(symbol=conv_comp, shape=shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define a function that constructs an Inception module based on `ConvFactory`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"2137pt\" height=\"1292pt\"\n",
       " viewBox=\"0.00 0.00 2137.36 1292.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 1288)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-1288 2133.36,-1288 2133.36,4 -4,4\"/>\n",
       "<!-- Previos Output -->\n",
       "<g id=\"node1\" class=\"node\"><title>Previos Output</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"522.359\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"522.359\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">Previos Output</text>\n",
       "</g>\n",
       "<!-- conv_in3a_1x1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>conv_in3a_1x1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"156.359,-878 62.3593,-878 62.3593,-820 156.359,-820 156.359,-878\"/>\n",
       "<text text-anchor=\"middle\" x=\"109.359\" y=\"-852.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"109.359\" y=\"-837.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_1x1&#45;&gt;Previos Output -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>conv_in3a_1x1&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M132.709,-811.289C149.194,-781.844 168.359,-738.83 168.359,-698 168.359,-698 168.359,-698 168.359,-138 168.359,-74.3877 380.124,-44.5695 476.08,-34.3233\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"127.704,-819.974 128.799,-809.063 130.201,-815.642 132.697,-811.31 132.697,-811.31 132.697,-811.31 130.201,-815.642 136.596,-813.557 127.704,-819.974 127.704,-819.974\"/>\n",
       "<text text-anchor=\"middle\" x=\"192.359\" y=\"-431.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_gamma -->\n",
       "<g id=\"node3\" class=\"node\"><title>bn_in3a_1x1_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"66.3593\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"66.3593\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_beta -->\n",
       "<g id=\"node4\" class=\"node\"><title>bn_in3a_1x1_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"178.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"178.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_moving_mean -->\n",
       "<g id=\"node5\" class=\"node\"><title>bn_in3a_1x1_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"290.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"290.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_moving_var -->\n",
       "<g id=\"node6\" class=\"node\"><title>bn_in3a_1x1_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"402.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"402.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1 -->\n",
       "<g id=\"node7\" class=\"node\"><title>bn_in3a_1x1</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"281.359,-1064 187.359,-1064 187.359,-1006 281.359,-1006 281.359,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"234.359\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;conv_in3a_1x1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;conv_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M177.086,-1032.15C123.06,-1026.91 45.3016,-1009.48 10.3593,-954 -3.37922,-932.188 -3.50589,-917.731 10.3593,-896 21.9295,-877.866 42.7003,-866.769 62.2392,-860.035\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"187.219,-1033.02 176.871,-1036.65 182.237,-1032.59 177.255,-1032.17 177.255,-1032.17 177.255,-1032.17 182.237,-1032.59 177.639,-1027.68 187.219,-1033.02 187.219,-1033.02\"/>\n",
       "<text text-anchor=\"middle\" x=\"68.8593\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_gamma -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M181.948,-1000.31C154.497,-982.66 121.776,-961.625 98.1672,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"190.658,-1005.91 179.813,-1004.28 186.452,-1003.2 182.246,-1000.5 182.246,-1000.5 182.246,-1000.5 186.452,-1003.2 184.68,-996.713 190.658,-1005.91 190.658,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_beta -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M215.032,-996.725C207.547,-982.291 199.146,-966.088 192.275,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"219.792,-1005.91 211.194,-999.1 217.491,-1001.47 215.189,-997.029 215.189,-997.029 215.189,-997.029 217.491,-1001.47 219.184,-994.957 219.792,-1005.91 219.792,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_moving_mean -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M253.687,-996.725C261.171,-982.291 269.573,-966.088 276.444,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"248.926,-1005.91 249.535,-994.957 251.228,-1001.47 253.53,-997.029 253.53,-997.029 253.53,-997.029 251.228,-1001.47 257.525,-999.1 248.926,-1005.91 248.926,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_moving_var -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M286.77,-1000.31C314.222,-982.66 346.942,-961.625 370.552,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"278.061,-1005.91 284.039,-996.713 282.267,-1003.2 286.473,-1000.5 286.473,-1000.5 286.473,-1000.5 282.267,-1003.2 288.906,-1004.28 278.061,-1005.91 278.061,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_1x1 -->\n",
       "<g id=\"node8\" class=\"node\"><title>relu_in3a_1x1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"690.359,-1174 596.359,-1174 596.359,-1116 690.359,-1116 690.359,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"643.359\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"643.359\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_1x1&#45;&gt;bn_in3a_1x1 -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>relu_in3a_1x1&#45;&gt;bn_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M585.968,-1128.85C504.542,-1107.34 357.014,-1068.39 281.566,-1048.47\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"596.042,-1131.51 585.225,-1133.3 591.208,-1130.23 586.374,-1128.95 586.374,-1128.95 586.374,-1128.95 591.208,-1130.23 587.523,-1124.6 596.042,-1131.51 596.042,-1131.51\"/>\n",
       "<text text-anchor=\"middle\" x=\"490.859\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3_reduce -->\n",
       "<g id=\"node9\" class=\"node\"><title>conv_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"486.359,-464 392.359,-464 392.359,-406 486.359,-406 486.359,-464\"/>\n",
       "<text text-anchor=\"middle\" x=\"439.359\" y=\"-438.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"439.359\" y=\"-423.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3_reduce&#45;&gt;Previos Output -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>conv_in3a_3x3_reduce&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M464.942,-397.083C479.7,-372.221 495.359,-338.052 495.359,-305 495.359,-305 495.359,-305 495.359,-138 495.359,-109.976 504.021,-79.0496 511.533,-57.4995\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"459.577,-405.802 460.985,-394.927 462.197,-401.544 464.817,-397.286 464.817,-397.286 464.817,-397.286 462.197,-401.544 468.65,-399.644 459.577,-405.802 459.577,-405.802\"/>\n",
       "<text text-anchor=\"middle\" x=\"519.359\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_gamma -->\n",
       "<g id=\"node10\" class=\"node\"><title>bn_in3a_3x3_reduce_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"402.359\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"402.359\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_beta -->\n",
       "<g id=\"node11\" class=\"node\"><title>bn_in3a_3x3_reduce_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"514.359\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_moving_mean -->\n",
       "<g id=\"node12\" class=\"node\"><title>bn_in3a_3x3_reduce_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"626.359\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"626.359\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_moving_var -->\n",
       "<g id=\"node13\" class=\"node\"><title>bn_in3a_3x3_reduce_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"738.359\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"738.359\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce -->\n",
       "<g id=\"node14\" class=\"node\"><title>bn_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"561.359,-650 467.359,-650 467.359,-592 561.359,-592 561.359,-650\"/>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-617.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;conv_in3a_3x3_reduce -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;conv_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M457.249,-611.023C417.256,-601.249 367.358,-581.108 346.359,-540 334.633,-517.044 332.76,-503.899 346.359,-482 356.623,-465.474 374.778,-454.704 392.34,-447.776\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"467.18,-613.291 456.429,-615.452 462.305,-612.178 457.431,-611.065 457.431,-611.065 457.431,-611.065 462.305,-612.178 458.433,-606.678 467.18,-613.291 467.18,-613.291\"/>\n",
       "<text text-anchor=\"middle\" x=\"400.859\" y=\"-562.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_gamma -->\n",
       "<g id=\"edge10\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M477.813,-584.759C461.23,-568.768 442.047,-550.27 427.173,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"485.225,-591.906 474.903,-588.204 481.626,-588.436 478.027,-584.965 478.027,-584.965 478.027,-584.965 481.626,-588.436 481.15,-581.726 485.225,-591.906 485.225,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_beta -->\n",
       "<g id=\"edge11\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M514.359,-581.813C514.359,-568.152 514.359,-553.009 514.359,-540.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"514.359,-591.906 509.859,-581.906 514.359,-586.906 514.359,-581.906 514.359,-581.906 514.359,-581.906 514.359,-586.906 518.859,-581.906 514.359,-591.906 514.359,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_moving_mean -->\n",
       "<g id=\"edge12\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M550.905,-584.759C567.489,-568.768 586.672,-550.27 601.545,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"543.494,-591.906 547.568,-581.726 547.093,-588.436 550.692,-584.965 550.692,-584.965 550.692,-584.965 547.093,-588.436 553.816,-588.204 543.494,-591.906 543.494,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_moving_var -->\n",
       "<g id=\"edge13\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M570.933,-593.392C603.578,-577.955 645.366,-558.054 682.359,-540 688.904,-536.806 695.859,-533.366 702.561,-530.027\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"561.546,-597.827 568.665,-589.486 566.067,-595.691 570.587,-593.555 570.587,-593.555 570.587,-593.555 566.067,-595.691 572.51,-597.624 561.546,-597.827 561.546,-597.827\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3_reduce -->\n",
       "<g id=\"node15\" class=\"node\"><title>relu_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"561.359,-802 467.359,-802 467.359,-744 561.359,-744 561.359,-802\"/>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-776.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-761.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce -->\n",
       "<g id=\"edge14\" class=\"edge\"><title>relu_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M514.359,-733.507C514.359,-707.451 514.359,-673.621 514.359,-650.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"514.359,-743.786 509.859,-733.786 514.359,-738.786 514.359,-733.786 514.359,-733.786 514.359,-733.786 514.359,-738.786 518.859,-733.786 514.359,-743.786 514.359,-743.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"541.859\" y=\"-693.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3 -->\n",
       "<g id=\"node16\" class=\"node\"><title>conv_in3a_3x3</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"561.359,-954 467.359,-954 467.359,-896 561.359,-896 561.359,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"514.359\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3&#45;&gt;relu_in3a_3x3_reduce -->\n",
       "<g id=\"edge15\" class=\"edge\"><title>conv_in3a_3x3&#45;&gt;relu_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M514.359,-885.507C514.359,-859.451 514.359,-825.621 514.359,-802.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"514.359,-895.786 509.859,-885.786 514.359,-890.786 514.359,-885.786 514.359,-885.786 514.359,-885.786 514.359,-890.786 518.859,-885.786 514.359,-895.786 514.359,-895.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"541.859\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_gamma -->\n",
       "<g id=\"node17\" class=\"node\"><title>bn_in3a_3x3_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"626.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"626.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_beta -->\n",
       "<g id=\"node18\" class=\"node\"><title>bn_in3a_3x3_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"738.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"738.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_moving_mean -->\n",
       "<g id=\"node19\" class=\"node\"><title>bn_in3a_3x3_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"850.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"850.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_moving_var -->\n",
       "<g id=\"node20\" class=\"node\"><title>bn_in3a_3x3_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"962.359\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"962.359\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3 -->\n",
       "<g id=\"node21\" class=\"node\"><title>bn_in3a_3x3</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"841.359,-1064 747.359,-1064 747.359,-1006 841.359,-1006 841.359,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"794.359\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;conv_in3a_3x3 -->\n",
       "<g id=\"edge16\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;conv_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M737.206,-1019.08C708.144,-1010.93 672.481,-1000.05 641.359,-988 608.731,-975.37 601.785,-969.379 570.359,-954 567.43,-952.567 564.419,-951.085 561.384,-949.585\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"747.192,-1021.85 736.353,-1023.52 742.373,-1020.52 737.555,-1019.18 737.555,-1019.18 737.555,-1019.18 742.373,-1020.52 738.757,-1014.84 747.192,-1021.85 747.192,-1021.85\"/>\n",
       "<text text-anchor=\"middle\" x=\"668.859\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_gamma -->\n",
       "<g id=\"edge17\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M741.948,-1000.31C714.497,-982.66 681.776,-961.625 658.167,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"750.658,-1005.91 739.813,-1004.28 746.452,-1003.2 742.246,-1000.5 742.246,-1000.5 742.246,-1000.5 746.452,-1003.2 744.68,-996.713 750.658,-1005.91 750.658,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_beta -->\n",
       "<g id=\"edge18\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M775.032,-996.725C767.547,-982.291 759.146,-966.088 752.275,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"779.792,-1005.91 771.194,-999.1 777.491,-1001.47 775.189,-997.029 775.189,-997.029 775.189,-997.029 777.491,-1001.47 779.184,-994.957 779.792,-1005.91 779.792,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_moving_mean -->\n",
       "<g id=\"edge19\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M813.687,-996.725C821.171,-982.291 829.573,-966.088 836.444,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"808.926,-1005.91 809.535,-994.957 811.228,-1001.47 813.53,-997.029 813.53,-997.029 813.53,-997.029 811.228,-1001.47 817.525,-999.1 808.926,-1005.91 808.926,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_moving_var -->\n",
       "<g id=\"edge20\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M846.77,-1000.31C874.222,-982.66 906.942,-961.625 930.552,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"838.061,-1005.91 844.039,-996.713 842.267,-1003.2 846.473,-1000.5 846.473,-1000.5 846.473,-1000.5 842.267,-1003.2 848.906,-1004.28 838.061,-1005.91 838.061,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3 -->\n",
       "<g id=\"node22\" class=\"node\"><title>relu_in3a_3x3</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"938.359,-1174 844.359,-1174 844.359,-1116 938.359,-1116 938.359,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"891.359\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"891.359\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3&#45;&gt;bn_in3a_3x3 -->\n",
       "<g id=\"edge21\" class=\"edge\"><title>relu_in3a_3x3&#45;&gt;bn_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M859.213,-1108.21C846.399,-1093.94 831.862,-1077.76 819.802,-1064.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"866.127,-1115.91 856.097,-1111.47 862.786,-1112.19 859.445,-1108.47 859.445,-1108.47 859.445,-1108.47 862.786,-1112.19 862.793,-1105.46 866.127,-1115.91 866.127,-1115.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"876.859\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_reduce -->\n",
       "<g id=\"node23\" class=\"node\"><title>conv_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"618.359,-168 524.359,-168 524.359,-110 618.359,-110 618.359,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"571.359\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"571.359\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_reduce&#45;&gt;Previos Output -->\n",
       "<g id=\"edge22\" class=\"edge\"><title>conv_in3a_double_3x3_reduce&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M554.48,-100.798C547.984,-86.4797 540.694,-70.412 534.703,-57.2072\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"558.613,-109.906 550.383,-102.659 556.547,-105.353 554.481,-100.8 554.481,-100.8 554.481,-100.8 556.547,-105.353 558.579,-98.9404 558.613,-109.906 558.613,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"574.359\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_gamma -->\n",
       "<g id=\"node24\" class=\"node\"><title>bn_in3a_double_3x3_reduce_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"684.359\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"684.359\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_beta -->\n",
       "<g id=\"node25\" class=\"node\"><title>bn_in3a_double_3x3_reduce_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"796.359\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"796.359\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_moving_mean -->\n",
       "<g id=\"node26\" class=\"node\"><title>bn_in3a_double_3x3_reduce_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"908.359\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"908.359\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_moving_var -->\n",
       "<g id=\"node27\" class=\"node\"><title>bn_in3a_double_3x3_reduce_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1020.36\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1020.36\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce -->\n",
       "<g id=\"node28\" class=\"node\"><title>bn_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"850.359,-278 756.359,-278 756.359,-220 850.359,-220 850.359,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"803.359\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;conv_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge23\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;conv_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M746.55,-229.429C724.871,-221.739 700.142,-212.226 678.359,-202 657.175,-192.056 634.633,-179.34 615.729,-168.015\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"756.275,-232.83 745.35,-233.777 751.556,-231.18 746.836,-229.529 746.836,-229.529 746.836,-229.529 751.556,-231.18 748.322,-225.281 756.275,-232.83 756.275,-232.83\"/>\n",
       "<text text-anchor=\"middle\" x=\"705.859\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_gamma -->\n",
       "<g id=\"edge24\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M764.95,-213.141C747.089,-196.931 726.315,-178.077 710.332,-163.572\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"772.404,-219.906 761.975,-216.518 768.702,-216.546 764.999,-213.186 764.999,-213.186 764.999,-213.186 768.702,-216.546 768.023,-209.853 772.404,-219.906 772.404,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_beta -->\n",
       "<g id=\"edge25\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M800.884,-209.813C799.999,-196.152 799.017,-181.009 798.195,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"801.538,-219.906 796.401,-210.218 801.215,-214.917 800.892,-209.927 800.892,-209.927 800.892,-209.927 801.215,-214.917 805.382,-209.636 801.538,-219.906 801.538,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_moving_mean -->\n",
       "<g id=\"edge26\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M837.987,-212.382C853.321,-196.611 870.964,-178.464 884.748,-164.286\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"830.673,-219.906 834.417,-209.599 834.158,-216.321 837.644,-212.736 837.644,-212.736 837.644,-212.736 834.158,-216.321 840.87,-215.873 830.673,-219.906 830.673,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_moving_var -->\n",
       "<g id=\"edge27\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M859.887,-219.954C890.767,-204.623 929.677,-185.287 964.359,-168 970.877,-164.751 977.819,-161.285 984.517,-157.938\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"850.575,-224.577 857.531,-216.1 855.054,-222.354 859.532,-220.131 859.532,-220.131 859.532,-220.131 855.054,-222.354 861.533,-224.161 850.575,-224.577 850.575,-224.577\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_reduce -->\n",
       "<g id=\"node29\" class=\"node\"><title>relu_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"892.359,-388 798.359,-388 798.359,-330 892.359,-330 892.359,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"845.359\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"845.359\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge28\" class=\"edge\"><title>relu_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M830.665,-320.215C825.312,-306.449 819.356,-291.134 814.376,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"834.434,-329.906 826.615,-322.217 832.622,-325.246 830.809,-320.586 830.809,-320.586 830.809,-320.586 832.622,-325.246 835.003,-318.955 834.434,-329.906 834.434,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"854.859\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_0 -->\n",
       "<g id=\"node30\" class=\"node\"><title>conv_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"897.359,-540 803.359,-540 803.359,-482 897.359,-482 897.359,-540\"/>\n",
       "<text text-anchor=\"middle\" x=\"850.359\" y=\"-514.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"850.359\" y=\"-499.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 96</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_0&#45;&gt;relu_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge29\" class=\"edge\"><title>conv_in3a_double_3x3_0&#45;&gt;relu_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M849.076,-471.507C848.208,-445.451 847.08,-411.621 846.296,-388.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"849.419,-481.786 844.588,-471.942 849.252,-476.789 849.086,-471.792 849.086,-471.792 849.086,-471.792 849.252,-476.789 853.583,-471.642 849.419,-481.786 849.419,-481.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"876.859\" y=\"-431.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_gamma -->\n",
       "<g id=\"node31\" class=\"node\"><title>bn_in3a_double_3x3_0_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"962.359\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"962.359\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_beta -->\n",
       "<g id=\"node32\" class=\"node\"><title>bn_in3a_double_3x3_0_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1074.36\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_moving_mean -->\n",
       "<g id=\"node33\" class=\"node\"><title>bn_in3a_double_3x3_0_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1186.36\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1186.36\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_moving_var -->\n",
       "<g id=\"node34\" class=\"node\"><title>bn_in3a_double_3x3_0_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1298.36\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1298.36\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0 -->\n",
       "<g id=\"node35\" class=\"node\"><title>bn_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"1121.36,-650 1027.36,-650 1027.36,-592 1121.36,-592 1121.36,-650\"/>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-617.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;conv_in3a_double_3x3_0 -->\n",
       "<g id=\"edge30\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;conv_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1017.72,-600.844C997.113,-593.259 973.861,-583.968 953.359,-574 932.9,-564.052 911.19,-551.335 893.003,-540.011\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1027.28,-604.316 1016.35,-605.135 1022.58,-602.61 1017.88,-600.905 1017.88,-600.905 1017.88,-600.905 1022.58,-602.61 1019.42,-596.675 1027.28,-604.316 1027.28,-604.316\"/>\n",
       "<text text-anchor=\"middle\" x=\"980.859\" y=\"-562.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_gamma -->\n",
       "<g id=\"edge31\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1037.81,-584.759C1021.23,-568.768 1002.05,-550.27 987.173,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1045.23,-591.906 1034.9,-588.204 1041.63,-588.436 1038.03,-584.965 1038.03,-584.965 1038.03,-584.965 1041.63,-588.436 1041.15,-581.726 1045.23,-591.906 1045.23,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_beta -->\n",
       "<g id=\"edge32\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1074.36,-581.813C1074.36,-568.152 1074.36,-553.009 1074.36,-540.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1074.36,-591.906 1069.86,-581.906 1074.36,-586.906 1074.36,-581.906 1074.36,-581.906 1074.36,-581.906 1074.36,-586.906 1078.86,-581.906 1074.36,-591.906 1074.36,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_moving_mean -->\n",
       "<g id=\"edge33\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1110.91,-584.759C1127.49,-568.768 1146.67,-550.27 1161.55,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1103.49,-591.906 1107.57,-581.726 1107.09,-588.436 1110.69,-584.965 1110.69,-584.965 1110.69,-584.965 1107.09,-588.436 1113.82,-588.204 1103.49,-591.906 1103.49,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_moving_var -->\n",
       "<g id=\"edge34\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1130.93,-593.392C1163.58,-577.955 1205.37,-558.054 1242.36,-540 1248.9,-536.806 1255.86,-533.366 1262.56,-530.027\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1121.55,-597.827 1128.67,-589.486 1126.07,-595.691 1130.59,-593.555 1130.59,-593.555 1130.59,-593.555 1126.07,-595.691 1132.51,-597.624 1121.55,-597.827 1121.55,-597.827\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_0 -->\n",
       "<g id=\"node36\" class=\"node\"><title>relu_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"1121.36,-802 1027.36,-802 1027.36,-744 1121.36,-744 1121.36,-802\"/>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-776.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-761.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0 -->\n",
       "<g id=\"edge35\" class=\"edge\"><title>relu_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1074.36,-733.507C1074.36,-707.451 1074.36,-673.621 1074.36,-650.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1074.36,-743.786 1069.86,-733.786 1074.36,-738.786 1074.36,-733.786 1074.36,-733.786 1074.36,-733.786 1074.36,-738.786 1078.86,-733.786 1074.36,-743.786 1074.36,-743.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"1101.86\" y=\"-693.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_1 -->\n",
       "<g id=\"node37\" class=\"node\"><title>conv_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"1121.36,-954 1027.36,-954 1027.36,-896 1121.36,-896 1121.36,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"1074.36\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 96</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_1&#45;&gt;relu_in3a_double_3x3_0 -->\n",
       "<g id=\"edge36\" class=\"edge\"><title>conv_in3a_double_3x3_1&#45;&gt;relu_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1074.36,-885.507C1074.36,-859.451 1074.36,-825.621 1074.36,-802.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1074.36,-895.786 1069.86,-885.786 1074.36,-890.786 1074.36,-885.786 1074.36,-885.786 1074.36,-885.786 1074.36,-890.786 1078.86,-885.786 1074.36,-895.786 1074.36,-895.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"1101.86\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_gamma -->\n",
       "<g id=\"node38\" class=\"node\"><title>bn_in3a_double_3x3_1_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1186.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1186.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_beta -->\n",
       "<g id=\"node39\" class=\"node\"><title>bn_in3a_double_3x3_1_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1298.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1298.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_moving_mean -->\n",
       "<g id=\"node40\" class=\"node\"><title>bn_in3a_double_3x3_1_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1410.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1410.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_moving_var -->\n",
       "<g id=\"node41\" class=\"node\"><title>bn_in3a_double_3x3_1_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1522.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1522.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1 -->\n",
       "<g id=\"node42\" class=\"node\"><title>bn_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"1289.36,-1064 1195.36,-1064 1195.36,-1006 1289.36,-1006 1289.36,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"1242.36\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;conv_in3a_double_3x3_1 -->\n",
       "<g id=\"edge37\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;conv_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1186.08,-1007.42C1174.68,-1001.43 1162.95,-994.821 1152.36,-988 1136.65,-977.886 1120.31,-965.271 1106.68,-954.065\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1195.12,-1012.08 1184.17,-1011.5 1190.68,-1009.79 1186.23,-1007.5 1186.23,-1007.5 1186.23,-1007.5 1190.68,-1009.79 1188.29,-1003.5 1195.12,-1012.08 1195.12,-1012.08\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.86\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_gamma -->\n",
       "<g id=\"edge38\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1223.03,-996.725C1215.55,-982.291 1207.15,-966.088 1200.27,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1227.79,-1005.91 1219.19,-999.1 1225.49,-1001.47 1223.19,-997.029 1223.19,-997.029 1223.19,-997.029 1225.49,-1001.47 1227.18,-994.957 1227.79,-1005.91 1227.79,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_beta -->\n",
       "<g id=\"edge39\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1261.69,-996.725C1269.17,-982.291 1277.57,-966.088 1284.44,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1256.93,-1005.91 1257.53,-994.957 1259.23,-1001.47 1261.53,-997.029 1261.53,-997.029 1261.53,-997.029 1259.23,-1001.47 1265.52,-999.1 1256.93,-1005.91 1256.93,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_moving_mean -->\n",
       "<g id=\"edge40\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1294.77,-1000.31C1322.22,-982.66 1354.94,-961.625 1378.55,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1286.06,-1005.91 1292.04,-996.713 1290.27,-1003.2 1294.47,-1000.5 1294.47,-1000.5 1294.47,-1000.5 1290.27,-1003.2 1296.91,-1004.28 1286.06,-1005.91 1286.06,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_moving_var -->\n",
       "<g id=\"edge41\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1299.49,-1015.72C1345.19,-1000.58 1410.63,-977.782 1466.36,-954 1473.16,-951.099 1480.3,-947.763 1487.11,-944.429\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1289.76,-1018.92 1297.85,-1011.52 1294.51,-1017.36 1299.26,-1015.79 1299.26,-1015.79 1299.26,-1015.79 1294.51,-1017.36 1300.66,-1020.07 1289.76,-1018.92 1289.76,-1018.92\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_1 -->\n",
       "<g id=\"node43\" class=\"node\"><title>relu_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"1090.36,-1174 996.359,-1174 996.359,-1116 1090.36,-1116 1090.36,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"1043.36\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"1043.36\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1 -->\n",
       "<g id=\"edge42\" class=\"edge\"><title>relu_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1099.65,-1113.45C1129.81,-1097.08 1166.55,-1077.14 1195.19,-1061.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1090.52,-1118.41 1097.16,-1109.68 1094.91,-1116.02 1099.31,-1113.64 1099.31,-1113.64 1099.31,-1113.64 1094.91,-1116.02 1101.46,-1117.59 1090.52,-1118.41 1090.52,-1118.41\"/>\n",
       "<text text-anchor=\"middle\" x=\"1182.86\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- avg_pool_in3a_pool -->\n",
       "<g id=\"node44\" class=\"node\"><title>avg_pool_in3a_pool</title>\n",
       "<polygon fill=\"#80b1d3\" stroke=\"black\" points=\"1518.36,-726 1424.36,-726 1424.36,-668 1518.36,-668 1518.36,-726\"/>\n",
       "<text text-anchor=\"middle\" x=\"1471.36\" y=\"-700.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"1471.36\" y=\"-685.8\" font-family=\"Times,serif\" font-size=\"14.00\">avg, 3x3/1</text>\n",
       "</g>\n",
       "<!-- avg_pool_in3a_pool&#45;&gt;Previos Output -->\n",
       "<g id=\"edge43\" class=\"edge\"><title>avg_pool_in3a_pool&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1441.55,-659.517C1424.46,-634.915 1406.36,-600.874 1406.36,-567 1406.36,-567 1406.36,-567 1406.36,-138 1406.36,-52.9143 752.761,-34.3013 569.574,-30.7556\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1447.52,-667.809 1438.03,-662.326 1444.6,-663.752 1441.68,-659.695 1441.68,-659.695 1441.68,-659.695 1444.6,-663.752 1445.33,-657.065 1447.52,-667.809 1447.52,-667.809\"/>\n",
       "<text text-anchor=\"middle\" x=\"1430.36\" y=\"-355.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_proj -->\n",
       "<g id=\"node45\" class=\"node\"><title>conv_in3a_proj</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"1681.36,-954 1587.36,-954 1587.36,-896 1681.36,-896 1681.36,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"1634.36\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"1634.36\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 32</text>\n",
       "</g>\n",
       "<!-- conv_in3a_proj&#45;&gt;avg_pool_in3a_pool -->\n",
       "<g id=\"edge44\" class=\"edge\"><title>conv_in3a_proj&#45;&gt;avg_pool_in3a_pool</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1608.06,-887.53C1575.6,-842.524 1521.1,-766.961 1491.67,-726.165\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1614.04,-895.834 1604.55,-890.356 1611.12,-891.779 1608.2,-887.723 1608.2,-887.723 1608.2,-887.723 1611.12,-891.779 1611.84,-885.091 1614.04,-895.834 1614.04,-895.834\"/>\n",
       "<text text-anchor=\"middle\" x=\"1622.36\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_gamma -->\n",
       "<g id=\"node46\" class=\"node\"><title>bn_in3a_proj_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1746.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1746.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_beta -->\n",
       "<g id=\"node47\" class=\"node\"><title>bn_in3a_proj_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1858.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1858.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_moving_mean -->\n",
       "<g id=\"node48\" class=\"node\"><title>bn_in3a_proj_moving_mean</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1970.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1970.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_moving_mean</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_moving_var -->\n",
       "<g id=\"node49\" class=\"node\"><title>bn_in3a_proj_moving_var</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"2082.36\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"2082.36\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_moving_var</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj -->\n",
       "<g id=\"node50\" class=\"node\"><title>bn_in3a_proj</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"1849.36,-1064 1755.36,-1064 1755.36,-1006 1849.36,-1006 1849.36,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"1802.36\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;conv_in3a_proj -->\n",
       "<g id=\"edge45\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;conv_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1745.79,-1007.73C1734.15,-1001.68 1722.16,-994.967 1711.36,-988 1695.78,-977.951 1679.65,-965.346 1666.19,-954.131\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1755.02,-1012.42 1744.07,-1011.9 1750.56,-1010.16 1746.11,-1007.89 1746.11,-1007.89 1746.11,-1007.89 1750.56,-1010.16 1748.15,-1003.88 1755.02,-1012.42 1755.02,-1012.42\"/>\n",
       "<text text-anchor=\"middle\" x=\"1738.86\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_gamma -->\n",
       "<g id=\"edge46\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1783.03,-996.725C1775.55,-982.291 1767.15,-966.088 1760.27,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1787.79,-1005.91 1779.19,-999.1 1785.49,-1001.47 1783.19,-997.029 1783.19,-997.029 1783.19,-997.029 1785.49,-1001.47 1787.18,-994.957 1787.79,-1005.91 1787.79,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_beta -->\n",
       "<g id=\"edge47\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1821.69,-996.725C1829.17,-982.291 1837.57,-966.088 1844.44,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1816.93,-1005.91 1817.53,-994.957 1819.23,-1001.47 1821.53,-997.029 1821.53,-997.029 1821.53,-997.029 1819.23,-1001.47 1825.52,-999.1 1816.93,-1005.91 1816.93,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_moving_mean -->\n",
       "<g id=\"edge48\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_moving_mean</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1854.77,-1000.31C1882.22,-982.66 1914.94,-961.625 1938.55,-946.448\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1846.06,-1005.91 1852.04,-996.713 1850.27,-1003.2 1854.47,-1000.5 1854.47,-1000.5 1854.47,-1000.5 1850.27,-1003.2 1856.91,-1004.28 1846.06,-1005.91 1846.06,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_moving_var -->\n",
       "<g id=\"edge49\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_moving_var</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1859.49,-1015.72C1905.19,-1000.58 1970.63,-977.782 2026.36,-954 2033.16,-951.099 2040.3,-947.763 2047.11,-944.429\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1849.76,-1018.92 1857.85,-1011.52 1854.51,-1017.36 1859.26,-1015.79 1859.26,-1015.79 1859.26,-1015.79 1854.51,-1017.36 1860.66,-1020.07 1849.76,-1018.92 1849.76,-1018.92\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_proj -->\n",
       "<g id=\"node51\" class=\"node\"><title>relu_in3a_proj</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"1334.36,-1174 1240.36,-1174 1240.36,-1116 1334.36,-1116 1334.36,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"1287.36\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"1287.36\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_proj&#45;&gt;bn_in3a_proj -->\n",
       "<g id=\"edge50\" class=\"edge\"><title>relu_in3a_proj&#45;&gt;bn_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1344.28,-1132.06C1446.74,-1110.58 1660.35,-1065.78 1755.21,-1045.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1334.43,-1134.13 1343.29,-1127.67 1339.32,-1133.1 1344.21,-1132.08 1344.21,-1132.08 1344.21,-1132.08 1339.32,-1133.1 1345.14,-1136.48 1334.43,-1134.13 1334.43,-1134.13\"/>\n",
       "<text text-anchor=\"middle\" x=\"1602.86\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat -->\n",
       "<g id=\"node52\" class=\"node\"><title>ch_concat_in3a_chconcat</title>\n",
       "<polygon fill=\"#fdb462\" stroke=\"black\" points=\"1013.36,-1284 919.359,-1284 919.359,-1226 1013.36,-1226 1013.36,-1284\"/>\n",
       "<text text-anchor=\"middle\" x=\"966.359\" y=\"-1251.3\" font-family=\"Times,serif\" font-size=\"14.00\">Concat</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_1x1 -->\n",
       "<g id=\"edge51\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M909.381,-1234.95C847.145,-1214.14 748.605,-1181.19 690.536,-1161.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"919.046,-1238.18 908.136,-1239.28 914.304,-1236.59 909.563,-1235.01 909.563,-1235.01 909.563,-1235.01 914.304,-1236.59 910.99,-1230.74 919.046,-1238.18 919.046,-1238.18\"/>\n",
       "<text text-anchor=\"middle\" x=\"851.859\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_3x3 -->\n",
       "<g id=\"edge52\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M940.952,-1217.41C931.181,-1203.34 920.183,-1187.51 911.032,-1174.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"946.85,-1225.91 937.45,-1220.26 943.998,-1221.8 941.146,-1217.69 941.146,-1217.69 941.146,-1217.69 943.998,-1221.8 944.842,-1215.13 946.85,-1225.91 946.85,-1225.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"960.859\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_double_3x3_1 -->\n",
       "<g id=\"edge53\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M992.444,-1217.41C1002.48,-1203.34 1013.77,-1187.51 1023.16,-1174.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"986.389,-1225.91 988.53,-1215.15 989.292,-1221.83 992.194,-1217.76 992.194,-1217.76 992.194,-1217.76 989.292,-1221.83 995.859,-1220.38 986.389,-1225.91 986.389,-1225.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"1037.86\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_proj -->\n",
       "<g id=\"edge54\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1022.97,-1234.95C1084.73,-1214.18 1182.47,-1181.29 1240.21,-1161.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1013.38,-1238.18 1021.42,-1230.73 1018.12,-1236.59 1022.86,-1234.99 1022.86,-1234.99 1022.86,-1234.99 1018.12,-1236.59 1024.29,-1239.26 1013.38,-1238.18 1013.38,-1238.18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1173.86\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f05a67cefd0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# @@@ AUTOTEST_OUTPUT_IGNORED_CELL\n",
    "def InceptionFactoryA(data, num_1x1, num_3x3red, num_3x3, num_d3x3red, num_d3x3, pool, proj, name):\n",
    "    # 1x1\n",
    "    c1x1 = ConvFactory(data=data, num_filter=num_1x1, kernel=(1, 1), name=('%s_1x1' % name))\n",
    "    # 3x3 reduce + 3x3\n",
    "    c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce')\n",
    "    c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), name=('%s_3x3' % name))\n",
    "    # double 3x3 reduce + double 3x3\n",
    "    cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1), name=('%s_double_3x3' % name), suffix='_reduce')\n",
    "    cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_0' % name))\n",
    "    cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_1' % name))\n",
    "    # pool + proj\n",
    "    pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(1, 1), pad=(1, 1), pool_type=pool, name=('%s_pool_%s_pool' % (pool, name)))\n",
    "    cproj = ConvFactory(data=pooling, num_filter=proj, kernel=(1, 1), name=('%s_proj' %  name))\n",
    "    # concat\n",
    "    concat = mx.symbol.Concat(*[c1x1, c3x3, cd3x3, cproj], name='ch_concat_%s_chconcat' % name)\n",
    "    return concat\n",
    "prev = mx.symbol.Variable(name=\"Previos Output\")\n",
    "in3a = InceptionFactoryA(prev, 64, 64, 64, 64, 96, \"avg\", 32, name=\"in3a\")\n",
    "mx.viz.plot_network(symbol=in3a, shape=shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can obtain the whole network by chaining multiple inception modulas. A complete example is available at [mxnet/example/image-classification/symbol_inception-bn.py](https://github.com/dmlc/mxnet/blob/master/example/image-classification/symbol_inception-bn.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group Multiple Symbols\n",
    "\n",
    "To construct neural networks with multiple loss layers, we can use `mxnet.sym.Group` to group multiple symbols together. The following example group two outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['softmax_output', 'regression_output']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = mx.sym.Variable('data')\n",
    "fc1 = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)\n",
    "net = mx.sym.Activation(data=fc1, name='relu1', act_type=\"relu\")\n",
    "out1 = mx.sym.SoftmaxOutput(data=net, name='softmax')\n",
    "out2 = mx.sym.LinearRegressionOutput(data=net, name='regression')\n",
    "group = mx.sym.Group([out1, out2])\n",
    "group.list_outputs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relations to NDArray\n",
    "\n",
    "As can be seen now, both Symbol and NDArray provide multi-dimensional array operations, such as `c=a+b` in MXNet. Sometimes users are confused which way to use. We briefly clarify the difference here, more detailed explanation are available [here](http://mxnet.readthedocs.io/en/latest/system/program_model.html). \n",
    "\n",
    "The `NDArray` provides an imperative programming alike interface, in which the computations are evaluated sentence by sentence. While `Symbol` is closer to declarative programming, in which we first declare the computation, and then evaluate with data. Examples in this category include regular expression and SQL.\n",
    "\n",
    "The pros for `NDArray`:\n",
    "- straightforward\n",
    "- easy to work with other language features (for loop, if-else condition, ..) and libraries (numpy, ..)\n",
    "- easy to step-by-step debug\n",
    "\n",
    "The pros for `Symbol`:\n",
    "- provides almost all functionalities of NDArray, such as +, \\*, sin, and reshape \n",
    "- provides a large number of neural network related operators such as Convolution, Activation, and BatchNorm\n",
    "- provides automatic differentiation \n",
    "- easy to construct and manipulate complex computations such as deep neural networks\n",
    "- easy to save, load, and visualization\n",
    "- easy for the backend to optimize the computation and memory usage\n",
    "\n",
    "We will show on the [mixed programming tutorial](./mixed.ipynb) how these two interfaces can be used together to develop a complete training program. This tutorial will focus on the usage of Symbol. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Symbol Manipulation *\n",
    "\n",
    "One important difference of `Symbol` comparing to `NDArray` is that, we first declare the computation, and then bind with data to run. \n",
    "\n",
    "In this section we introduce the functions to manipulate a symbol directly. But note that, most of them are wrapped nicely by the [`mx.module`](./module.ipynb). One can skip this section safely.  \n",
    "\n",
    "### Shape Inference\n",
    "For each symbol, we can query its inputs (or arguments) and outputs. We can also inference the output shape by given the input shape, which facilitates memory allocation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': {'a': (2L, 3L), 'b': (2L, 3L)},\n",
       " 'output': {'_plus0_output': (2L, 3L)}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arg_name = c.list_arguments()  # get the names of the inputs\n",
    "out_name = c.list_outputs()    # get the names of the outputs\n",
    "arg_shape, out_shape, _ = c.infer_shape(a=(2,3), b=(2,3))  \n",
    "{'input' : dict(zip(arg_name, arg_shape)), \n",
    " 'output' : dict(zip(out_name, out_shape))}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bind with Data and Evaluate\n",
    "The symbol `c` we constructed declares what computation should be run. To evaluate it, we need to feed arguments, namely free variables, with data first. We can do it by using the `bind` method, which accepts device context and a `dict` mapping free variable names to `NDArray`s as arguments and returns an executor. The executor provides method `forward` for evaluation and attribute `outputs` to get all results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of outputs = 1\n",
      "the first output = \n",
      "[[ 2.  2.  2.]\n",
      " [ 2.  2.  2.]]\n"
     ]
    }
   ],
   "source": [
    "ex = c.bind(ctx=mx.cpu(), args={'a' : mx.nd.ones([2,3]), \n",
    "                                'b' : mx.nd.ones([2,3])})\n",
    "ex.forward()\n",
    "print 'number of outputs = %d\\nthe first output = \\n%s' % (\n",
    "           len(ex.outputs), ex.outputs[0].asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the same symbol on GPU with different data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.,  5.,  5.,  5.],\n",
       "       [ 5.,  5.,  5.,  5.],\n",
       "       [ 5.,  5.,  5.,  5.]], dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_gpu = c.bind(ctx=mx.gpu(), args={'a' : mx.nd.ones([3,4], mx.gpu())*2,\n",
    "                                    'b' : mx.nd.ones([3,4], mx.gpu())*3})\n",
    "ex_gpu.forward()\n",
    "ex_gpu.outputs[0].asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Load and Save\n",
    "\n",
    "Similar to NDArray, we can either serialize a `Symbol` object by using `pickle`, or use `save` and `load` directly. Different to the binary format chosen by `NDArray`, `Symbol` uses the more readable json format for serialization. The `tojson` method returns the json string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"nodes\": [\n",
      "    {\n",
      "      \"op\": \"null\", \n",
      "      \"name\": \"a\", \n",
      "      \"inputs\": []\n",
      "    }, \n",
      "    {\n",
      "      \"op\": \"null\", \n",
      "      \"name\": \"b\", \n",
      "      \"inputs\": []\n",
      "    }, \n",
      "    {\n",
      "      \"op\": \"elemwise_add\", \n",
      "      \"name\": \"_plus0\", \n",
      "      \"inputs\": [[0, 0, 0], [1, 0, 0]]\n",
      "    }\n",
      "  ], \n",
      "  \"arg_nodes\": [0, 1], \n",
      "  \"node_row_ptr\": [0, 1, 2, 3], \n",
      "  \"heads\": [[2, 0, 0]], \n",
      "  \"attrs\": {\"mxnet_version\": [\"int\", 901]}\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(c.tojson())\n",
    "c.save('symbol-c.json')\n",
    "c2 = mx.symbol.load('symbol-c.json')\n",
    "c.tojson() == c2.tojson()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customized Symbol *\n",
    "\n",
    "Most operators such as `mx.sym.Convolution` and `mx.sym.Reshape` are implemented in C++ for better performance. MXNet also allows users to write new operators using any frontend language such as Python. It often makes the developing and debugging much easier. \n",
    "\n",
    "To implement an operator in Python, we just need to define the two computation methods `forward` and `backward` with several methods for querying the properties, such as `list_arguments` and `infer_shape`. \n",
    "\n",
    "`NDArray` is the default type of arguments in both `forward` and `backward`. Therefore we often also implement the computation with `NDArray` operations. To show the flexibility of MXNet, however, we will demonstrate an implementation of the `softmax` layer using NumPy. Though a NumPy based operator can be only run on CPU and also lose some optimizations which can be applied on NDArray, it enjoys the rich functionalities provided by NumPy.\n",
    "\n",
    "We first create a subclass of `mx.operator.CustomOp` and then define `forward` and `backward`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Softmax(mx.operator.CustomOp):\n",
    "    def forward(self, is_train, req, in_data, out_data, aux):\n",
    "        x = in_data[0].asnumpy()\n",
    "        y = np.exp(x - x.max(axis=1).reshape((x.shape[0], 1)))\n",
    "        y /= y.sum(axis=1).reshape((x.shape[0], 1))\n",
    "        self.assign(out_data[0], req[0], mx.nd.array(y))\n",
    "\n",
    "    def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n",
    "        l = in_data[1].asnumpy().ravel().astype(np.int)\n",
    "        y = out_data[0].asnumpy()\n",
    "        y[np.arange(l.shape[0]), l] -= 1.0\n",
    "        self.assign(in_grad[0], req[0], mx.nd.array(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we use `asnumpy` to convert the `NDArray` inputs into `numpy.ndarray`. Then using `CustomOp.assign` to assign the results back to `mxnet.NDArray` based on the value of req, which could be \"over write\" or \"add to\". \n",
    "\n",
    "Next we create a subclass of `mx.operator.CustomOpProp` for querying the properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# register this operator into MXNet by name \"softmax\"\n",
    "@mx.operator.register(\"softmax\")\n",
    "class SoftmaxProp(mx.operator.CustomOpProp):\n",
    "    def __init__(self):\n",
    "        # softmax is a loss layer so we don’t need gradient input\n",
    "        # from layers above. \n",
    "        super(SoftmaxProp, self).__init__(need_top_grad=False)\n",
    "    \n",
    "    def list_arguments(self):\n",
    "        return ['data', 'label']\n",
    "\n",
    "    def list_outputs(self):\n",
    "        return ['output']\n",
    "\n",
    "    def infer_shape(self, in_shape):\n",
    "        data_shape = in_shape[0]\n",
    "        label_shape = (in_shape[0][0],)\n",
    "        output_shape = in_shape[0]\n",
    "        return [data_shape, label_shape], [output_shape], []\n",
    "\n",
    "    def create_operator(self, ctx, shapes, dtypes):\n",
    "        return Softmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use `mx.sym.Custom` with the register name to use this operator\n",
    "```python\n",
    "net = mx.symbol.Custom(data=prev_input, op_type='softmax')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Usages *\n",
    "\n",
    "### Type Cast\n",
    "MXNet uses 32-bit float in default. Sometimes we want to use a lower precision data type for better accuracy-performance trade-off. For example, The Nvidia Tesla Pascal GPUs (e.g. P100) have improved 16-bit float performance, while GTX Pascal GPUs (e.g. GTX 1080) are fast on 8-bit integers. \n",
    "\n",
    "We can use the `mx.sym.Cast` operator to convert the data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': [<type 'numpy.float32'>], 'output': [<type 'numpy.float16'>]}\n",
      "{'input': [<type 'numpy.int32'>], 'output': [<type 'numpy.uint8'>]}\n"
     ]
    }
   ],
   "source": [
    "a = mx.sym.Variable('data')\n",
    "b = mx.sym.Cast(data=a, dtype='float16')\n",
    "arg, out, _ = b.infer_type(data='float32')\n",
    "print({'input':arg, 'output':out})\n",
    "\n",
    "c = mx.sym.Cast(data=a, dtype='uint8')\n",
    "arg, out, _ = c.infer_type(data='int32')\n",
    "print({'input':arg, 'output':out})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable Sharing\n",
    "\n",
    "Sometimes we want to share the contents between several symbols. This can be simply done by bind these symbols with the same array. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6.,  6.,  6.],\n",
       "       [ 6.,  6.,  6.]], dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = mx.sym.Variable('a')\n",
    "b = mx.sym.Variable('b')\n",
    "c = mx.sym.Variable('c')\n",
    "d = a + b * c\n",
    "\n",
    "data = mx.nd.ones((2,3))*2\n",
    "ex = d.bind(ctx=mx.cpu(), args={'a':data, 'b':data, 'c':data})\n",
    "ex.forward()\n",
    "ex.outputs[0].asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Readings\n",
    "\n",
    "- [Use torch operators in MXNet](http://mxnet.dmlc.ml/en/latest/how_to/torch.html)\n",
    "- [Use Caffe operators in MXNet](http://dmlc.ml/mxnet/2016/07/29/use-caffe-operator-in-mxnet.html)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
